Papers with large scale models
Knowledge Enhanced Contextual Word Representations (D19-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith
| Challenge: | Existing methods to embed knowledge bases into large pre-training models do not contain any explicit grounding to real world entities and are difficult to recover factual knowledge. |
| Approach: | They propose a method to embed multiple knowledge bases (KBs) into large pretrained models with a Knowledge Attention and Recontextualization mechanism. |
| Outcome: | The proposed model improves perplexity, ability to recall facts and word sense disambiguation. |
Recipes for Building an Open-Domain Chatbot (2021.eacl-main)
Copied to clipboard
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, Jason Weston
| Challenge: | Existing work shows that scaling models in the number of parameters and the size of the data they are trained on gives improved results, but other factors are important. |
| Approach: | They propose to build open-domain chatbots that can be scaled to improve their performance . they use a blend of cognitive and cognitive skills to build a model that combines these skills . |
| Outcome: | The proposed models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. |
Data and Parameter Scaling Laws for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that supervised neural machine translation models scale like a power law with the amount of training data and number of non-embedding parameters in the model. |
| Approach: | They show that cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and number of non-embedding parameters in the model. |
| Outcome: | The proposed model can predict BLEU and ROI of labeling data in low-resource language pairs. |
Learning under Label Proportions for Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using a set of data samples with ground truth labels, learning from label proportions is infeasible in many cases. |
| Approach: | They propose a setup where the data is provided in an aggregate form called bags and only the proportion of samples belonging to each class as the ground truth. |
| Outcome: | The proposed model achieves better results than baselines in almost 87% of the configurations. |